[AI] Add neural restore lighttable module for AI denoise and upscale#20523
[AI] Add neural restore lighttable module for AI denoise and upscale#20523andriiryzhkov wants to merge 3 commits intodarktable-org:masterfrom
Conversation
|
Excellent! I was looking forward to it :) |
|
How do these models fit in with the "acceptable" model conversation? If we merge a module that requires a model that's not "acceptable", then darktable is "endorsing"/requiring the non acceptable model. |
These models easily meet criteria for open-source AI. Please, see details here: |
|
@esq4: Good point about disk usage. The current approach (import as grouped image) is intentional — it gives you a denoised/upscaled source that you can further develop in darkroom and compare via grouping. This is conceptually different from export, which is a final output step. More flexibility would definitely be helpful here. What I think can be added as extra parameters:
As for export-time denoising (like nind-denoise) — that's a different use case but a valid one. It could be added to the export module down the road as a complementary feature. |
|
BSRGAN only meets open source tooling and in the limitations it says If I look at some of the other models (SAM) they require data destruction in 3 months. Does that mean I have to destroy my edits? It also says no commercial use, so if I sell one of my images am I in violation? I'm sorry, but this is a minefield. Somehow we need to decide a quick way to determine if a model is acceptable. Do we use the OSAID, and if so what MOF? It seems Class I, Open Science, is fine. However Class II, Open Tooling, seems to come with lots of limitations/questions. If we decide to use Class II, how are we going to communicate the limitations to the users? Don't expect them to read, we already know how well that works. |
You are referring to clause 7 of SA-1B DATASET RESEARCH LICENSE. But that is a requirement for the training dataset, not the model. You can use that dataset for your research, for training your own model, but you can't keep it indefinitely. What's the problem?
Absolutely not. This is a conclusion from your statement above that is not true.
Also no. The user is using the model and not the training dataset.
I'm also sorry, but this is NOT a minefield. Wanna see photos of what a minefield actually is? :)
Too vague. What is "acceptable"? Why should we decide what is acceptable for users and not the users themselves? |
|
Thank you for this PR. It works for me. The denoise and upscale functions work. I use the nind and bsrgan models from preferences -> AI after downloading them. If I test this PR on top of git master branch I get fallback to CPU and can not activate NVIDIA CUDA. The denose and, even more so, upscale, are very heavy using CPU and I would say that a GPU is essential (not so for AI masks). Workaround: I found a couple of preliminary issues:
|
|
I'm getting:
when running this on my RX 6700 XT with HSA_OVERRIDE_GFX_VERSION=10.3.0 also bsrgan model won't download throwing an error about a missing integrity check as it couldn't download the chksum |
I guess you have recognized that there is some problem within the dt dev community about exactly this point. And yes, if we as devs decide - we don't want something for whatever reason that may be - it's not a user decision at all. Maybe minefield was not the perfect wording. But you just should accept that for some of the long time devs the "who did and how a model was made" is absolutely critical. For me personally a razor-sharp yes-or-no. |
|
CPU seems to work. But I think it should export to the default export folder. And will it in future also be possible to integrate it right into the normal export workflow? Also without the override and AI set to off in the setting it crashes darktable: I think it should just disable AI and make the checkbox grey or only allow CPU |
|
Note that as for the AI masking there is no models installed and downloaded automatically. All this stays in the hand of the end-users, so everyone can decide depending on its sensibility. I recognize that we have different perception about AI and I would never allow any model installed and/or delivered by default. Even the default build from source has no AI support, one need to pass a specific option to enable this. I really think the current state is respecting everyone choice. |
|
@piratenpanda: thank you for testing and feedback. Regarding error you got. When running neural restore with MIGraphX provider on RX 6700 XT, the model compilation fails because MIGraphX can't handle dynamic shapes in concat operations with
That's interesting. It works on my side. Thinking of possible causes - can you double-check that in
Agree on disabling AI preferences when AI is disabled. This feature along with with proper AI actions block when AI is disabled implemented in PR #20534. It also should fix startup crash when AI is disabled. I would really appreciate your help testing it further. |
😢
If the dev specifies a model necessary to run the module, it's not a user choice, it's a dev/darktable choice. For example I could take the GIMP lua script and replace gimp with photoshop and then add it to the lua-scripts repository and put the "blame" on the user for running it. The open source community would see that as darktable endorsing/requiring/encouraging the use of photoshop. But... Someone could build a script that lets you specify the external executable, such as ext_editor.lua, and you can decide for yourself what executables you want to run even if it's photoshop, capture one, dxo, etc, etc, etc. That is a user choice and darktable has no say in the user's decision. The ONLY way the model can be a user choice is if darktable has NO SAY in the decision. |
Indeed it was the old one still. Changing to the right one it works fine
with the mentioned PR it does not crash anymore |
Fixed - output TIFF now embeds linear Rec.709 ICC profile and source EXIF.
Added collapsible "output parameters" section: bit depth (8/16/32, default 16), catalog toggle, and output directory with darktable variable support (e.g. |
|
Windows CI failing with: |
Are we going to give the user any information about the model's licensing or are we putting that all on the user? Or are we expecting them to go digging to find out for themselves? If we don't at least point them at the information how are they supposed to make an informed choice. Perhaps we include the OSAID rating and the MOF rating with a brief explanation (MOF I - conforms, MOF 2 - some exceptions, etc) and links? |
Probably a good idea, yes! |
So we are expecting this as the last edit step so done after all processing made to the image? |
This is a third and improved part of the original PR #20322.
Summary
neural_restore) that provides AI-based image restoration using ONNX backend modelsai_models.json)Details
USE_AI=ON(cmake option, default OFF)Fixes: #19310